Tag
10 articles
OpenAI has found that around 30% of tasks in the popular SWE-Bench Pro benchmark are broken, leading the company to withdraw its endorsement of the test.
OpenAI introduces GeneBench-Pro, a new benchmark testing AI performance in genomics and biological research using complex, real-world datasets.
Claude Opus 4.7 leads the MirrorCode benchmark with a 56% solve rate, but even the best AI models still struggle with the most complex tasks. Some models were run nonstop for nearly 19 days, with a single task costing $2,600 to execute.
Even the best AI models struggle with real knowledge work, successfully completing just 3 percent of complex tasks in a new benchmark.
OpenAI introduces LifeSciBench, an expert-authored benchmark for evaluating AI systems in real-world life science research tasks. The platform aims to advance AI's role in scientific discovery by providing a standardized framework for assessing AI capabilities in biomedical research.
LifeSciBench is a comprehensive benchmark developed by OpenAI that evaluates AI models on real-life science research tasks, focusing on reasoning and decision-making rather than simple recall. It uses expert-authored rubrics to assess how well AI systems can handle complex scientific workflows.
A new AI benchmark reveals that models confidently solve math problems that have no solution, exposing a key gap in their reasoning capabilities.
The ARC-AGI-3 benchmark challenges AI systems to match untrained human performance in interactive environments, with no frontier model achieving more than 1% success. The test strips away AI's typical advantages, exposing a gap in reasoning and adaptability.
ServiceNow Research introduces EnterpriseOps-Gym, a high-fidelity benchmark to evaluate agentic planning in realistic enterprise environments. The tool addresses key challenges like long-horizon planning and access controls.
OpenAI plans to retire the SWE-bench Verified benchmark, citing flaws that undermine its validity as a coding performance measure. The move highlights concerns about memorization in AI model evaluations.
